docs: confirm review pipeline already routes through orchestrator/free, not NIM directly - #1884

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands.}

Current-head documentation blocker (freshly refetched at 0847c4424e7067b0dedd9ffcdbc9cb62780b897d over protected main@6d7fbebec8aec31d88a30a36e71ca5b3925d241d):

The statement that the review pipeline is already fully routed through contextual-orchestrator and needs no code change is too broad for Strix. The logical model identifier is now contextual-orchestrator/orchestrator/free, but protected .github/workflows/strix.yml still injects five raw provider secrets into scripts/ci/contextual_orchestrator_review_sidecar.sh; that script requires at least one provider secret, clones CO source at runtime, and performs provider discovery locally. The Strix job also remains harden-runner egress-policy: audit.

NewsDOM #682's exact predecessor run 32638076034 / job 97190711585 proves why this distinction matters: the old workflow directly selected NVIDIA, received HTTP 429, and attempted NVIDIA/OpenAI fallbacks. Current main improves the logical route but does not yet establish the requested released gateway-token-only, block-mode egress boundary.

Please keep this PR Draft and do not characterize the dependency as complete. Canonical implementation/dependency tracking is now:

• central: Migrate Strix/OpenCode/Noema/autofix sidecar steps to the orchestrator-free-sidecar composite action #1759, updated with exact current evidence and migration order;
• contextual-orchestrator: issue fix(security): fail closed when dependency-review compare is unavailable #1041 comment 5550412102, requiring an immutable released gateway/client/schema/egress contract.

This is a source/architecture finding, not a request for a source-neutral rerun or a NewsDOM change.

Confirmed — independently re-verified all three points against exact file:line evidence (.github/workflows/strix.yml:745-758 for the five injected provider secrets, scripts/ci/contextual_orchestrator_review_sidecar.sh lines 51-63/91-98 for the runtime clone-and-build, lines 108-109 and 316-328 for in-process local discovery, strix.yml:365-367 for harden-runner egress-policy: audit) and they all held up.

Pushed 50de5f63 correcting docs/product-technical-gap-baseline.md's entry and its matching CHANGELOG.md entry: the model-selection/logical-routing layer (opencode.jsonc, opencode-review-dispatch.yml) is still accurately described as correct, but the "Confirmed already implemented; no code change needed" framing is removed and replaced with an explicit split — the sidecar/egress infrastructure layer that Strix (and the other three #1759 consumers) actually run on is not yet complete, with the same evidence you cited and the same #1759 / contextual-orchestrator#1041 comment 5550412102 tracking links. Keeping this PR Draft as requested.

Generated by Claude Code

Taking this out of Draft.

The hold requested in comment 5550414804 on 2026-09-05 was tied to the "already implemented, no code change needed" framing; that framing was replaced by the two-layer split in 50de5f63 the same morning, and the correction has stood unchallenged since. Two changes since then, both pushed as b4d45fd2..cbb61c5a:

1. Merged main@fb2ae81d into the branch (no conflicts; the CHANGELOG entry stays in ## [Unreleased]).
2. Added a dated 2026-09-06 follow-up to the same gap-baseline entry: the orchestrator/free retry-stacking root cause (contextual-orchestrator#1081), the ModelClient.single_attempt_transport() fix, the hard-coded sidecar pin that kept it out of production until #1951, the rule for which runs count as post-advance evidence, and the first post-pin measurement (.github#1661 run 34008191123: vendoring contextual-orchestrator @ 414f2297… live; preflight 0 of 12 routes ready, review request never made). It is recorded as "fixed, delivery confirmed, effect unconfirmed", separate from the still-open sidecar/egress gap under #1759.

Still documentation-only. Doc contract tests: 44 passed; git diff --check clean.

Marking ready also makes this the first ready-for-review head on this repository pushed after the pin advance, so its noema-review / opencode-review / strix runs should log vendoring contextual-orchestrator @ 414f2297… — whatever they show is exactly the evidence the follow-up says is still missing, and I will fold it back into the entry. If the hold should stay for a reason beyond the framing, say so here and I will re-draft.

Generated by Claude Code

noema-review on cbb61c5a (job 101455078239, 09:09–09:11Z) failed at sidecar provisioning, before any review request. Pin 414f2297… vendored; preflight probed 6, ready 0, rejected 6, deferred 0, skipped 18 of candidate_count 24 — 429 from both NVIDIA keys on deepseek-v4-flash and deepseek-v4-pro and from two OpenRouter free routes, all within 90 ms, then #1949's account skip retired the rest. This is preflight-fail (capacity) per #1948, identical to three other boots in the same 45 minutes (#1187 08:29Z, #1411 noema-review 08:55Z and strix 09:08Z). Nothing in a documentation-only diff is exercised by it and no fix exists in this PR's scope; not re-running while the pool reads zero ready.

Pushed b35947ea meanwhile, which is what those measurements are for. The 2026-09-06 follow-up in docs/product-technical-gap-baseline.md now closes the retry-stacking item instead of leaving it unverified:

• Effect confirmed. #1946 run 34008655765 (first Noema run past route preflight on the new pin) served each gateway-preflight attempt with exactly two 90 s tries on its one ready route — 180 s per gateway request against 540 s under the old pin — and the three post-advance Strix scans show attempt=1/1 throughout while pushing real work through the pool. contextual-orchestrator#1081 is closed.
• What remains is not that defect, and is now written down as three separate items: capacity (#1948, with the four-run census above), contextual-orchestrator#1082 (a 90 s passthrough timeout surfacing as 500 internal_error with _record_failure unreachable, so the same silent route is re-selected — 176 timeouts across those three Strix scans, ≈ 4.4 of their 5.6 runner-hours), and #1948's open owner decision on whether a preflight probe deadline is a policy value distinct from the inference deadline.

Measurements are a second lane's postings on #1948; I re-read the four job logs cited here directly. Doc contract tests: 44 passed; git diff --check clean.

Generated by Claude Code

noema-review failed at 15:37Z on 9c010fcb. It is not this PR's — this PR changes two Markdown files and no code — and the finding is interesting enough that I have folded it into the document the PR maintains rather than only reporting it.

From the job (run 34035522521, job 101501520756, step started 15:14:08Z, failed 15:37:54Z):

Noema gateway transport failed: HTTPError: HTTP Error 502: Bad Gateway;
caller attempts=1, duration=1424.1s, phase=response_error,
served_model=deepseek-ai/deepseek-v4-flash-0731

This is a fourth shape, distinct from all three residuals this PR already records:

phase=response_error means a response arrived carrying an error status, not a socket expiring — a different event from the TimeoutError in #1053's own 90.054 s noema measurement. And caller attempts=1 bounds the caller: the gateway owns repair and failover, as its own warning line on the next line of the log says, so those 23.7 minutes are the gateway walking its pool internally before classifying.

Consequence worth stating plainly: closing capacity and the raw-500 path will not by itself account for this. Whatever bounded this request, it was not the ModelClient 90 s default that both #1053 and my own residual (iii) treat as the operative limit.

Honest limit. This is the job log. The noema-sidecar-evidence artifact (9992218398, 2366 bytes) holds the sidecar stderr and preflight JSON that would give the gateway's internal attempt count and how the 23.7 minutes was distributed. I have not read it, so I claim no per-attempt breakdown — only that a served route plus a classified 502 plus a 23.7-minute wall clock is a real, current combination.

Recorded as residual (iv) in docs/product-technical-gap-baseline.md and in the changelog, in ace6f347. Full suite 2956 passed, 1 skipped, 21 subtests.

No fix pushed and no re-run spent, because there is nothing here for a docs-only diff to fix and a re-run would re-enter the same gateway. The upstream fixes remain contextual-orchestrator#1053 and #1082, both unmerged and both other lanes' work; my standing commitment to advance this repository's sidecar pin the moment the later of them lands is unchanged.

Generated by Claude Code

Correction to my 15:45Z comment. I found I could download the noema-sidecar-evidence artifacts after all (GET /repos/…/actions/artifacts/{id}/zip, ≈2.3 KB each), and reading them refutes two of the three claims that comment made. Both retractions are written into residual (iv) in a7ce345f rather than edited away.

1. Nothing was served. served_model names the last route attempted. Both runs end provider_attempt_failed … deepseek-v4-flash-0731 … error_type=TimeoutError → circuit_failure … failures=1.0 threshold=3 → request_failed status=502 code=provider_connection_error. I read a field name as an outcome.
2. The 90 s default is operative, so this was never evidence against contextual-orchestrator#1053. caller attempts=1 bounds the caller only: the gateway ran 24 internal attempts summing to ≈11,500 s against a 1,424 s wall clock — 8–9× concurrency, they race — and four sit at 89.5–92 s in both runs. I have corrected that on #1053 directly, since I had told that lane their premise was contradicted.

What survives, and it is the useful part. The attempt durations are bimodal: besides the four at ~90 s and six failing under 10 s, 11 of 24 attempts here and 12 of 24 on #1187 ran past 600 s, to 1333.7 s, with medians of 478.3 s and 631.3 s. No 90 s bound explains that second population. So the merge-order consequence runs opposite to what I implied: removing the implicit timeout converts the ~90 s attempts into unbounded waits and leaves the long ones untouched, which on this evidence should make these runs longer. Stated as a two-sample prediction, not a proven regression.

Two readings that support the other lanes rather than complicate them: the breaker is told on this path (circuit_failure 13×, circuit_opened 2×), matching #1082's scoping of its defect to the passthrough walk; and its re-selection concentration shows here too, with deepseek-v4-flash-0731 taking 23 of ~40 attempts across both NVIDIA keys despite the breaker opening twice. Preflight read ready_count 6, rejected 8, deferred 2, skipped 4 of 24 candidates, so capacity is now ruled out by artifact rather than by inference.

The general lesson I have taken from this and recorded: when an entry says "I did not read the evidence, so I claim no breakdown", that is a reason to go get the evidence, not a licence to reason from the summary line. The caller line was self-consistent and wrong.

Full suite 2956 passed, 1 skipped, 21 subtests.

Generated by Claude Code

opencode-review on 396b4dee — not this PR's change

What failed. run 34044467903 / job 101528130806, 10 s wall clock:

##[error]No APPROVED or CHANGES_REQUESTED from opencode-agent on the current head.
The dispatch workflow will rerun this failed job after publishing an authenticated exact-head verdict.

Why it is not this PR's. The dispatch step succeeded — the required job did its whole job and then fail-closed by design, because no opencode-agent review exists yet at commit_id == 396b4dee. This is the documented two-run sequence, not a defect in the diff: the required job requests the review and fails closed; opencode-review-dispatch.yml then publishes the authenticated exact-head verdict and reruns this job. The diff here is docs/product-technical-gap-baseline.md + CHANGELOG.md only — it cannot influence whether an opencode-agent review exists.

The dispatch this job requested is run 34050888549, created 2026-09-06T18:10:50Z, still queued with its single validate-pr-metadata job unstarted.

Why it is still queued (measured 2026-09-06T18:1x against this repo's Actions API):

• status=queued → 150 runs, oldest waiting 39 min.
• status=in_progress → 18 runs, of which 14 are Strix Security Scan, aged 97–381 min.
• Six of those fourteen are non-PR #event@ scans on successive main commits — c232ca, dd0b96, ee5567, 6e014c, 9aad23, ad0779 — alive simultaneously. Five are superseded by 49eb9e70.

That last row is a consequence of a deliberate, contract-pinned decision, not an accident: strix.yml's workflow-level group falls through to github.run_id for events with no PR number, and tests/test_required_workflow_queue_contract.py::test_strix_serializes_provider_evidence_per_repository_and_pr pins it ("non-PR events use a unique run id"). A unique key makes cancel-in-progress: true a no-op for that event class, so every push to main adds one more multi-hour runner holder rather than retiring the previous one. Recording the occupancy cost here as measurement; the contract change itself is not this PR's scope and overlaps work merged today in #1964/#1970/#1975/#1979.

Re-run. Not spending one. The failure message names the mechanism that performs the re-run (opencode-review-dispatch.yml reruns this failed job after publishing the verdict), and that dispatch is the run currently queued. A manual re-run before the dispatch completes would re-enter the same queue and fail-close identically in 10 s. For the same reason this PR is not being pushed to right now: a synchronize event would enqueue a fresh opencode-review / noema-review / strix / CodeQL / Semgrep / Security Scan set into the 150-deep queue this comment is about.

Same class, same window, other repositories — OpenCode Review Dispatch runs completed failure in the last two hours for fast-mlsirm#1764, naruon#1531, naruon#1244, contextual-orchestrator, life-os#247, and .github#1967, with .github#834, .github#1644, bandscope#866 and bandscope#1076 still queued. This PR is not the variable.

This PR stays watched until it is green.

Generated by Claude Code

Correction to my previous comment: this is not a wait-state

Above I described opencode-review on 396b4dee as the designed two-run sequence, with dispatch 34050888549 merely queued behind a saturated runner pool. The first half is right; the conclusion I drew from it is wrong, and it changes what a reader should expect.

That dispatch will not publish a verdict when it dequeues. It will be rejected at validate-pr-metadata before any review job runs, because the required workflow sends repos/.../dispatches with a minted OpenCode App token — actor = sender = opencode-agent[bot] — while the receiving gate's vars.OPENCODE_REPOSITORY_DISPATCH_ACTOR allowlist contains only github-actions[bot]. Three failures sampled across three repositories today stop at exactly that step with coverage-source-tree / coverage-evidence / opencode-review all skipped, and exactly one opencode-review-dispatch.yml run has concluded success since 2026-08-31 — the one whose actor was github-actions[bot].

So the queue depth I measured is real but it is not what blocks this check, and waiting will not clear it. Full evidence and the reason the remedy is the allowlist rather than the sender (a pull_request_target job must not get contents: write) is on #1927, which already tracks this: #1927 (comment)

vars.OPENCODE_REPOSITORY_DISPATCH_ACTOR is an Actions variable, so no diff on this PR can carry the fix, and this session cannot read or write it (403 through the agent proxy). This PR's opencode-review and noema-review stay red until that owner action lands. Everything in the diff itself — docs/product-technical-gap-baseline.md and CHANGELOG.md — is unaffected by it. Still watched.

Generated by Claude Code

Correction: noema-review is not blocked by the dispatch allowlist

My previous comment ended "This PR's opencode-review and noema-review stay red until that owner action lands." The opencode-review half stands. The noema-review half is wrong, and I am correcting it rather than leaving the two coupled: noema-review never sends a repository_dispatch, so vars.OPENCODE_REPOSITORY_DISPATCH_ACTOR cannot be what fails it. Fixing that allowlist will not turn this check green.

What actually fails. Job 101521704729, 2026-09-06T17:33:26Z → 18:14:14Z (41 min). Every step through Provision contextual-orchestrator review sidecar succeeds; Prepare Noema model verdict is the failure, and the sidecar log's last line is:

request_failed status=502 code=provider_connection_error

The preflight was healthy — this is not a gateway-unreachable run (artifact noema-sidecar-evidence, id 9994541963):

"gateway": {"attempts": 1, "endpoint": "chat/completions", "finish_reason": "stop", "status": "ready"}
"candidate_count": 24, "ready_count": 6, "deferred_count": 2, "rejected_count": 8, "skipped_count": 4

All six ready routes are one provider family:

Nothing else was available to fall back to: provider_discovery_failed provider=bytez code=http_status_500, and both OpenRouter free routes deferred on HTTPError / 429. The eight rejects are NVIDIA 404s on retired models plus timeouts.

Then every attempt on those six routes timed out at ~90 s. Eight measured intervals from provider_attempt to provider_attempt_failed … error_type=TimeoutError: 90.1, 90.1, 90.1, 90.2, 90.1, 90.1, 90.0, 90.1 s — the ModelClient socket idle timeout, not a total-elapsed deadline. The run then walks circuit_failure → circuit_opened(reset 30 s) → circuit_cleared → retry, exhausts attempts=3 on the last route, and ends 502.

So this check's blocker is free-pool provider-family availability, which #1915 tracks as the acceptance item ("eliminate orchestrator/free provider-family SPOF"). I have posted this run as evidence there. It is still not this PR's change — the diff is docs/product-technical-gap-baseline.md + CHANGELOG.md — and per the standing policy I am not converting elapsed inference time into a timeout cap here. Re-run not spent: a re-run selects from the same six-route single-family pool.

Standing summary for this PR: opencode-review needs the allowlist owner action (#1927); noema-review needs free-pool diversity (#1915). Neither is fixable in this diff. Still watched.

Generated by Claude Code

The blocker named above is gone

My earlier comments on this PR said opencode-review here would "stay red until that owner action lands." It landed: vars.OPENCODE_REPOSITORY_DISPATCH_ACTOR now admits opencode-agent[bot], so App-token dispatches are no longer rejected at validate-dispatch. codeql-scan-dispatch.yml has gone from 0 successes across 2352 runs to 10, all with that actor, and opencode-review-dispatch.yml has its first new success since 2026-09-05. Evidence: #1927 (comment)

This PR is now on 767e740c — main@c9052e60 merged in, plus the 2026-09-07 gap-baseline entry recording that outage and its fix, the free-pool single-family collapse (#1915), and three claims of mine retracted on #2000 under an independent count. Local gates before pushing: 2975 passed / 1 skipped, interrogate 100%, git diff --check clean. Checks on the new head are running with no failures so far.

noema-review is a separate matter and is unaffected by the allowlist fix — it never dispatches, and its blocker is free-pool provider-family availability (#1915). Still watched.

Generated by Claude Code